Data processing and online reconstruction

In the upcoming upgrades for Run 3 and 4, the LHC will significantly increase Pb--Pb and pp interaction rates. This goes along with upgrades of all experiments, ALICE, ATLAS, CMS, and LHCb, related to both the detectors and the computing. The online processing farms must employ faster, more efficient reconstruction algorithms to cope with the increased data rates, and data compression factors must increase to fit the data in the affordable capacity for permanent storage. Due to different operating conditions and aims, the experiments follow different approaches, but there are several common trends like more extensive online computing and the adoption of hardware accelerators. This paper gives an overview and compares the data processing approaches and the online computing farms of the LHC experiments today in Run 2 and for the upcoming LHC Run 3 and 4.Comment: 6 pages, 0 figures, contribution to LHCP2018 conferenc

Overview of online and offline reconstruction in ALICE for LHC Run 3

In LHC Run 3, ALICE will increase the data taking rate significantly to 50 kHz continuous readout of minimum bias Pb--Pb collisions. The reconstruction strategy of the online-offline computing upgrade foresees a first synchronous online reconstruction stage during data taking enabling detector calibration, and a posterior calibrated asynchronous reconstruction stage. The main challenges include processing and compression of 50 times more events per second than in Run 2, identification of removable TPC tracks and hits not used for physics, tracking of TPC data in continuous readout, the TPC space-charge distortion calibrations, and in general running more reconstruction steps online compared to Run 2. ALICE will leverage GPUs to facilitate the synchronous processing with the available resources. For the best GPU resource utilization, we plan to offload also several steps of the asynchronous reconstruction to the GPU. In order to be vendor independent, we support CUDA, OpenCL, and HIP, and we maintain a common C++ source code that also runs on the CPU. We will give an overview of the global reconstruction and tracking strategy, a comparison of the performance on CPU and different GPU models. We will discuss the scaling of the reconstruction with the input data size, as well as estimates of the required resources in terms of memory and processing power.Comment: 8 pages, 3 figures, proceedings of Connecting the Dots 2020 Worksho

GPU-based reconstruction and data compression at ALICE during LHC Run 3

In LHC Run 3, ALICE will increase the data taking rate significantly to 50 kHz continuous read out of minimum bias Pb-Pb collisions. The reconstruction strategy of the online offline computing upgrade foresees a first synchronous online reconstruction stage during data taking enabling detector calibration, and a posterior calibrated asynchronous reconstruction stage. The significant increase in the data rate poses challenges for online and offline reconstruction as well as for data compression. Compared to Run 2, the online farm must process 50 times more events per second and achieve a higher data compression factor. ALICE will rely on GPUs to perform real time processing and data compression of the Time Projection Chamber (TPC) detector in real time, the biggest contributor to the data rate. With GPUs available in the online farm, we are evaluating their usage also for the full tracking chain during the asynchronous reconstruction for the silicon Inner Tracking System (ITS) and Transition Radiation Detector (TRD). The software is written in a generic way, such that it can also run on processors on the WLCG with the same reconstruction output. We give an overview of the status and the current performance of the reconstruction and the data compression implementations on the GPU for the TPC and for the global reconstruction.Comment: 7 pages, 4 figures, proceedings of CHEP 2019 conferenc

GPU-based Online Track Reconstruction for the ALICE TPC in Run 3 with Continuous Read-Out

In LHC Run 3, ALICE will increase the data taking rate significantly to 50 kHz continuous read-out of minimum bias Pb-Pb collisions. The reconstruction strategy of the online-offline computing upgrade foresees a first synchronous online reconstruction stage during data taking enabling detector calibration and data compression, and a posterior calibrated asynchronous reconstruction stage. Many new challenges arise, among them continuous TPC read-out, more overlapping collisions, no a priori knowledge of the primary vertex and of location-dependent calibration in the synchronous phase, identification of low-momentum looping tracks, and sophisticated raw data compression. The tracking algorithm for the Time Projection Chamber (TPC) will be based on a Cellular Automaton and the Kalman filter. The reconstruction shall run online, processing 50 times more collisions per second than today, while yielding results comparable to current offline reconstruction. Our TPC track finding leverages the potential of hardware accelerators via the OpenCL and CUDA APIs in a shared source code for CPUs and GPUs for both reconstruction stages. We give an overview of the status of Run 3 tracking including performance on processors and GPUs and achieved compression ratios.Comment: 8 pages, 7 figures, contribution to CHEP 2018 conferenc

Track Reconstruction in the ALICE TPC using GPUs for LHC Run 3

In LHC Run 3, ALICE will increase the data taking rate significantly to continuous readout of 50 kHz minimum bias Pb-Pb collisions. The reconstruction strategy of the online offline computing upgrade foresees a first synchronous online reconstruction stage during data taking enabling detector calibration, and a posterior calibrated asynchronous reconstruction stage. We present a tracking algorithm for the Time Projection Chamber (TPC), the main tracking detector of ALICE. The reconstruction must yield results comparable to current offline reconstruction and meet the time constraints like in the current High Level Trigger (HLT), processing 50 times as many collisions per second as today. It is derived from the current online tracking in the HLT, which is based on a Cellular automaton and the Kalman filter, and we integrate missing features from offline tracking for improved resolution. The continuous TPC readout and overlapping collisions pose new challenges: conversion to spatial coordinates and the application of time- and location dependent calibration must happen in between of track seeding and track fitting while the TPC occupancy increases five-fold. The huge data volume requires a data reduction factor of 20, which imposes additional requirements: the momentum range must be extended to identify low-pt looping tracks and a special refit in uncalibrated coordinates improves the track model entropy encoding. Our TPC track finding leverages the potential of hardware accelerators via the OpenCL and CUDA APIs in a shared source code for CPUs, GPUs, and both reconstruction stages. Porting more reconstruction steps like the remainder of the TPC reconstruction and tracking for other detectors will shift the computing balance from traditional processors to GPUs.Comment: 13 pages, 10 figures, proceedings to Connecting The Dots Workshop, Seattle, 201

Fast TPC Online Tracking on GPUs and Asynchronous Data Processing in the ALICE HLT to facilitate Online Calibration

ALICE (A Large Heavy Ion Experiment) is one of the four major experiments at the Large Hadron Collider (LHC) at CERN, which is today the most powerful particle accelerator worldwide. The High Level Trigger (HLT) is an online compute farm of about 200 nodes, which reconstructs events measured by the ALICE detector in real-time. The HLT uses a custom online data-transport framework to distribute data and workload among the compute nodes. ALICE employs several calibration-sensitive subdetectors, e.g. the TPC (Time Projection Chamber). For a precise reconstruction, the HLT has to perform the calibration online. Online-calibration can make certain Offline calibration steps obsolete and can thus speed up Offline analysis. Looking forward to ALICE Run III starting in 2020, online calibration becomes a necessity. The main detector used for track reconstruction is the TPC. Reconstructing the trajectories in the TPC is the most compute-intense step during event reconstruction. Therefore, a fast tracking implementation is of great importance. Reconstructed TPC tracks build the basis for the calibration making a fast online-tracking mandatory. We present several components developed for the ALICE High Level Trigger to perform fast event reconstruction and to provide features required for online calibration. As first topic, we present our TPC tracker, which employs GPUs to speed up the processing, and which bases on a Cellular Automaton and on the Kalman filter. Our TPC tracking algorithm has been successfully used in 2011 and 2012 in the lead-lead and the proton-lead runs. We have improved it to leverage features of newer GPUs and we have ported it to support OpenCL, CUDA, and CPUs with a single common source code. This makes us vendor independent. As second topic, we present framework extensions required for online calibration. ...Comment: 8 pages, 6 figures, contribution to CHEP 2015 conferenc

BioEM: GPU-accelerated computing of Bayesian inference of electron microscopy images

In cryo-electron microscopy (EM), molecular structures are determined from large numbers of projection images of individual particles. To harness the full power of this single-molecule information, we use the Bayesian inference of EM (BioEM) formalism. By ranking structural models using posterior probabilities calculated for individual images, BioEM in principle addresses the challenge of working with highly dynamic or heterogeneous systems not easily handled in traditional EM reconstruction. However, the calculation of these posteriors for large numbers of particles and models is computationally demanding. Here we present highly parallelized, GPU-accelerated computer software that performs this task efficiently. Our flexible formulation employs CUDA, OpenMP, and MPI parallelization combined with both CPU and GPU computing. The resulting BioEM software scales nearly ideally both on pure CPU and on CPU+GPU architectures, thus enabling Bayesian analysis of tens of thousands of images in a reasonable time. The general mathematical framework and robust algorithms are not limited to cryo-electron microscopy but can be generalized for electron tomography and other imaging experiments

Online Calibration of the TPC Drift Time in the ALICE High Level Trigger

ALICE (A Large Ion Collider Experiment) is one of four major experiments at the Large Hadron Collider (LHC) at CERN. The High Level Trigger (HLT) is a compute cluster, which reconstructs collisions as recorded by the ALICE detector in real-time. It employs a custom online data-transport framework to distribute data and workload among the compute nodes. ALICE employs subdetectors sensitive to environmental conditions such as pressure and temperature, e.g. the Time Projection Chamber (TPC). A precise reconstruction of particle trajectories requires the calibration of these detectors. Performing the calibration in real time in the HLT improves the online reconstructions and renders certain offline calibration steps obsolete speeding up offline physics analysis. For LHC Run 3, starting in 2020 when data reduction will rely on reconstructed data, online calibration becomes a necessity. Reconstructed particle trajectories build the basis for the calibration making a fast online-tracking mandatory. The main detectors used for this purpose are the TPC and ITS (Inner Tracking System). Reconstructing the trajectories in the TPC is the most compute-intense step. We present several improvements to the ALICE High Level Trigger developed to facilitate online calibration. The main new development for online calibration is a wrapper that can run ALICE offline analysis and calibration tasks inside the HLT. On top of that, we have added asynchronous processing capabilities to support long-running calibration tasks in the HLT framework, which runs event-synchronously otherwise. In order to improve the resiliency, an isolated process performs the asynchronous operations such that even a fatal error does not disturb data taking. We have complemented the original loop-free HLT chain with ZeroMQ data-transfer components. [...]Comment: 8 pages, 10 figures, proceedings to 2016 IEEE-NPSS Real Time Conferenc